Indicator for voice-based communications

ABSTRACT

Systems, methods, and devices for outputting indications regarding voice-based interactions are described. A first speech-controlled device detects spoken audio corresponding to recipient information. The first device captures the audio and sends audio data corresponding to the captured audio to a server. The server determines a second speech-controlled device of the recipient and sends a signal to the recipient&#39;s second speech-controlled device representing a message is forthcoming. The recipient&#39;s second speech-controlled device outputs and indication representing a message is forthcoming.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims the benefit ofpriority of, U.S. Non-Provisional patent application Ser. No.15/254,600, entitled INDICATOR FOR VOICE-BASED COMMUNICATIONS, filedSep. 1, 2016, the contents of which is expressly incorporated herein byreference in its entirety.

BACKGROUND

Speech recognition systems have progressed to the point where humans caninteract with computing devices by relying on speech. Such systemsemploy techniques to identify the words spoken by a human user based onthe various qualities of a received audio input. Speech recognitioncombined with natural language understanding processing techniquesenable speech-based user control of a computing device to perform tasksbased on the user's spoken commands. The combination of speechrecognition and natural language understanding processing techniques isreferred to herein as speech processing. Speech processing may alsoinvolve converting a user's speech into text data which may then beprovided to various text-based software applications.

Speech processing may be used by computers, hand-held devices, telephonecomputer systems, kiosks, and a wide variety of other devices to improvehuman-computer interactions.

BRIEF DESCRIPTION OF DRAWINGS

For a more complete understanding of the present disclosure, referenceis now made to the following description taken in conjunction with theaccompanying drawings.

FIG. 1A illustrates a system for altering voice-based interactions viaspeech-controlled devices.

FIG. 1B illustrates a system for outputting signals to a user duringmessaging via speech-controlled devices.

FIG. 2 is a conceptual diagram of a speech processing system.

FIG. 3 is a conceptual diagram of a multi-domain architecture approachto natural language understanding.

FIG. 4 illustrates data stored and associated with user profiles.

FIGS. 5A through 5D are a signal flow diagram illustrating alteration ofvoice-based interaction via speech-controlled devices.

FIGS. 6A and 6B are a signal flow diagram illustrating alteration ofvoice-based interaction via speech-controlled devices.

FIG. 7 is a signal flow diagram illustrating alteration of a voice-basedinteraction via speech-controlled devices.

FIGS. 8A and 8B are a signal flow diagram illustrating the output ofsignaling via user interfaces of speech-controlled devices.

FIG. 9 is a signal flow diagram illustrating the output of signaling viauser interfaces of speech-controlled devices.

FIGS. 10A through 10C illustrate example signals output to a user via aspeech-controlled device.

FIGS. 11A and 11B illustrate an example signal output to a user via aspeech-controlled device.

FIG. 12 illustrates an example signal output to a user via aspeech-controlled device.

FIG. 13 is a block diagram conceptually illustrating example componentsof speech-controlled devices according to embodiments of the presentdisclosure.

FIG. 14 is a block diagram conceptually illustrating example componentsof a server according to embodiments of the present disclosure.

FIG. 15 illustrates an example of a computer network for use with thesystem of the present disclosure.

DETAILED DESCRIPTION

Automatic speech recognition (ASR) is a field of computer science,artificial intelligence, and linguistics concerned with transformingaudio data associated with speech into text representative of thatspeech. Similarly, natural language understanding (NLU) is a field ofcomputer science, artificial intelligence, and linguistics concernedwith enabling computers to derive meaning from text input containingnatural language. ASR and NLU are often used together as part of aspeech processing system.

ASR and NLU can be computationally expensive. That is, significantcomputing resources may be needed to process ASR and NLU processingwithin a reasonable time frame. Because of this, a distributed computingenvironment may be used when performing speech processing. A typicalsuch distributed environment may involve a local or other type of clientdevice having one or more microphones being configured to capture soundsfrom a user speaking and convert those sounds into an audio signal. Theaudio signal may then be sent to a remote device for further processing,such as converting the audio signal into an ultimate command. Thecommand may then be executed by a combination of remote and user devicesdepending on the command itself.

In certain configurations, a speech processing system may be configuredto communicate spoken messages between devices. That is, a first devicemay capture an utterance commanding the system to send a message to arecipient associated with a second device. In response, the user of thesecond device may speak an utterance that is captured by the seconddevice, and then sent to the system for processing to send a messageback to the user of the first device. In this manner a speech controlledsystem may facilitate spoken messaging between devices.

One drawback to such messaging, however, is that for each spokeninteraction with the system, a user may need to speak both a wakeword(to “wake up” a user device) as well as a recipient of the message, sothe system knows how to route the message included in the utterance.Such a traditional configuration may add friction to the interactionbetween the user(s) and the system, particularly when two users areexchanging multiple messages between them.

The present disclosure provides techniques for altering voice-basedinteractions via speech-controlled devices. Speech-controlled devicescapture audio, including wakeword portions and payload portions, forsending to a server to relay messages between speech-controlled devices.In response to determining the occurrence of a communication alterationtrigger, such as repeated messages between the same two devices, thesystem may automatically change a mode of a speech-controlled device,such as no longer requiring a wakeword, no longer requiring anindication of a desired recipient, or automatically connecting the twospeech-controlled devices in a voice-chat mode. When the mode of thespeech-controlled device is changed, the system may use differentprotocols to govern how messages and other data are exchanged betweendevices of the system. For example, when the system switches fromexchanging voice messages between devices to initiating a synchronouscall (e.g., phone call) between devices, the system may stop using amessaging protocol and activate or invoke a real-time protocol (e.g., aVoice over Internet Protocol (VoIP)). In response to determining theoccurrence of further communication altering triggers, the system mayinitiate a real-time, synchronous call between the speech-controlleddevices. Various examples of communication altering triggers andhandling by the system are illustrated below. Communication alterationtriggers as described herein may be system determined based on thesatisfaction of configured thresholds. That is, the system may beconfigured to alter communication exchanges without receiving anexplicit indication from a user to do so.

The present disclosure also provides techniques for outputting visual(or audio, haptic, etc.) indications regarding voice-based interactions.Such an indication may provide feedback using a first device's userinterface, the feedback indicating that a second device's inputcomponent (e.g., microphone) is in the process of receiving a userinput, such as a reply to a message sent from the first user's device.After the server sends message content to a recipient'sspeech-controlled device, the server may receive an indication from therecipient's speech-controlled device that the device is detectingspeech. In response, the server then causes a visual indication to beoutput by the first speech-controlled device, with the visual indicationrepresenting the recipient-speech controlled device is detecting speech.As such, it should be appreciated that the visual indication may be usedto keep users of speech-controlled devices from “talking over” eachother (i.e., prevent users of the speech-controlled devices fromsimultaneously speaking messages).

FIG. 1A shows a system 100 configured to alter voice-based interactionsbetween speech-controlled devices. Although FIG. 1A, and lowerfigures/discussion, illustrate the operation of the system 100 in aparticular order, the steps described may be performed in a differentorder (as well as certain steps removed or added) without departing fromthe intent of the disclosure. As shown in FIG. 1A, the system 100 mayinclude one or more speech-controlled devices 110 a and 110 b local to afirst user 5 and a second user 7, respectively. The system 100 alsoincludes one or more networks 199 and one or more servers 120 connectedto the devices 110 a and 110 b across network(s) 199. The server(s) 120(which may be one or more different physical devices) may be capable ofperforming traditional speech processing (such as ASR, NLU, queryparsing, etc.) as described herein. A single server may be capable ofperforming all speech processing or multiple servers may combine toperform the speech processing. Further, the server(s) 120 may beconfigured to execute certain commands, such as answering queries spokenby the first user 5 and/or second user 7. In addition, certain speechdetection or command execution functions may be performed by the devices110 a and 110 b.

As shown in FIG. 1A, the user 5 may speak an utterance (represented byinput audio 11). The input audio 11 may be captured by one or moremicrophones 103 a of the device 110 a and/or a microphone array (notillustrated) separated from the device 110 a. The microphone array maybe connected to the device 110 a such that when the input audio 11 isreceived by the microphone array, the microphone array sends audio datacorresponding to the input audio 11 to the device 110 a. Alternatively,the microphone array may be connected to a companion application of amobile computing device (not illustrated), such as a smart phone,tablet, etc. In this example, when the microphone array captures theinput audio 11, the microphone array sends audio data corresponding tothe input audio 11 to the companion application, which forwards theaudio data to the device 110 a. If the device 110 a captures the inputaudio 11, the device 110 a may convert the input audio 11 into audiodata and send the audio data to the server(s) 120. Alternatively, if thedevice 110 a receives audio data corresponding to the input audio 11from the microphone array or companion application, the device 110 a maysimply forward the received audio data to the server(s) 120.

The server(s) 120 originally communicates messages betweenspeech-controlled devices in response to receiving (150) audio dataincluding a wakeword portion and a payload portion. The payload portionmay include recipient information and message content. Communication ofthe messages as such may occur through use of a message domain andassociated protocol(s) as described in detail herein. The server 120communicates messages as such until the server 120 determines (152) theoccurrence of a first communication alteration trigger. Illustrativecommunication alternation triggers include whether a threshold number ofmessage exchanges between the first speech-controlled device 110 a andthe second speech-controlled device 110 b is met or exceeded, athreshold number of message exchanges occurring with a threshold amountof time, or users of both of the speech-controlled devices 110 a/110 bsimultaneously being with threshold distances of their respectivedevice. After determining the occurrence of the first communicationalteration trigger, the server 120 then communicates (154) messagesbetween the same speech-controlled devices in response to receivingaudio data including payload data (e.g., message content data).Communication of messages may occur through use of a messaging domainand associated protocol(s) as described in detail herein. The server 120communicates messages using the messaging domain until the server 120determines (156) the occurrence of a second communication alterationtrigger. After determining the occurrence of the second communicationalteration trigger, the server 120 then initiates (158) a real-time callbetween the speech-controlled devices. Initiating the real-time call mayinvolve use of a real-time call domain and associated real-timeprotocol(s) as described in detail herein. A real-time communicationsession/call may involve the passing of audio data between devices asthe audio data is received (within operational parameters).

Alternatively, after determination (152) of the first communicationalteration trigger, the server 120 may go straight to initiating (158)the real-time call. This may occur under different configuredcircumstances, such as when the communication alteration trigger ispremised upon a certain recipient. For example, a user profileassociated with the originating speech-controlled device 110 a mayindicate that communications with “mom” are to occur via real-timecalls. Thus, if the original message is intended for “mom,” the server120 may facilitate a real-time call in response to determining therecipient of the first message is “mom.”

According to various embodiments, the server 120 may cause one or bothof the speech-controlled devices to output visual indications usingrespective device user interfaces, with the visual indicationsrepresenting which domain is being used to exchangecommunications/messages. For example, a light on the speech-controlleddevice may emit a blue color when a wakeword is needed, may emit a greencolor when a wakeword is no longer needed, and may emit a yellow colorwhen the real-time call is facilitated.

In addition to altering voice-based exchanges to voice-based calls asdescribed herein above, the above teachings may be used in the contextof video communications. For example, if two individuals are exchangingvideo messages, the techniques herein described may be used to alter theexchange of video messages to a video call. In another example, ifindividuals are determined to be in fields of views of cameras whileexchanging voice-based messages, the system may be configured to alterthe communications to a video call based on the individuals being in thecameras' fields of view. Thus, teachings below regarding detectingspeech, capturing audio, or the like may also be applied to detectingvideo, capturing video, or the like.

Each speech-controlled device may have more than one user. The system100 may use voice-based speaker IDs or user IDs to identify a speaker ofcaptured audio. Each speaker ID or user ID may be a voice signature thatenables the system to determine the user of the device that is speaking.This is beneficial because it allows the system to alter communicationsas described herein when communication alteration triggers involve asingle user of a device. The speaker ID or user ID may be used todetermine who is speaking and automatically identify the speaker's userprofile for purposes of subsequent processing. For example, if a firstuser of a device speaks a message, and thereafter a second user of thedevice speaks a message, the system is able to distinguish the two usersbased on voice signature, thereby preventing the system from determininga single communication alteration trigger based on the messages spokenby different users.

FIG. 1B illustrates a system for outputting signals, via device userinterfaces, during messaging to indicate that responsive speech is beingdetected by a recipient's device. As shown in FIG. 1B, the systemreceives (160) input audio from a first speech-controlled device 110 a.The system then determines (162) that the input audio corresponds tomessage content for the second speech-controlled device 110 b. Thesystem then sends (164) the message content to the secondspeech-controlled device 110 b. The system then detects (166) speechusing the second speech-controlled device 110 b and causes (168) anindicator to be output by the first speech-controlled device 110 a,where the indicator represents speech is being detected by the seconddevice, where the speech may be in response to the message content, andthus notifies a user of the first speech-controlled device 110 a that areply may be imminent. The indicator may be visual, audible, or haptic.In an example, the indicator may be visual for a video enabled device.

Further details of escalating a voice-based interaction are discussedbelow, following a discussion of the overall speech processing system ofFIG. 2. FIG. 2 is a conceptual diagram of how a spoken utterance istraditionally processed, allowing a system to capture and executecommands spoken by a user, such as spoken commands that may follow awakeword. The various components illustrated may be located on a same ordifferent physical devices. Communication between various componentsillustrated in FIG. 2 may occur directly or across a network 199. Anaudio capture component, such as a microphone 103 of device 110,captures audio 11 corresponding to a spoken utterance. The device 110,using a wakeword detection module 220, then processes the audio, oraudio data corresponding to the audio, to determine if a keyword (suchas a wakeword) is detected in the audio. Following detection of awakeword, the device sends audio data 111 corresponding to theutterance, to a server 120 that includes an ASR module 250. The audiodata 111 may be output from an acoustic front end (AFE) 256 located onthe device 110 prior to transmission. Or the audio data 111 may be in adifferent form for processing by a remote AFE 256, such as the AFE 256located with the ASR module 250.

The wakeword detection module 220 works in conjunction with othercomponents of the device 110, for example a microphone (not pictured) todetect keywords in audio 11. For example, the device 110 may convertaudio 11 into audio data, and process the audio data with the wakeworddetection module 220 to determine whether speech is detected, and if so,if the audio data comprising speech matches an audio signature and/ormodel corresponding to a particular keyword.

The device 110 may use various techniques to determine whether audiodata includes speech. Some embodiments may apply voice activitydetection (VAD) techniques. Such techniques may determine whether speechis present in an audio input based on various quantitative aspects ofthe audio input, such as the spectral slope between one or more framesof the audio input; the energy levels of the audio input in one or morespectral bands; the signal-to-noise ratios of the audio input in one ormore spectral bands; or other quantitative aspects. In otherembodiments, the device 110 may implement a limited classifierconfigured to distinguish speech from background noise. The classifiermay be implemented by techniques such as linear classifiers, supportvector machines, and decision trees. In still other embodiments, HiddenMarkov Model (HMM) or Gaussian Mixture Model (GMM) techniques may beapplied to compare the audio input to one or more acoustic models inspeech storage, which acoustic models may include models correspondingto speech, noise (such as environmental noise or background noise), orsilence. Still other techniques may be used to determine whether speechis present in the audio input.

Once speech is detected in the audio received by the device 110 (orseparately from speech detection), the device 110 may use the wakeworddetection module 220 to perform wakeword detection to determine when auser intends to speak a command to the device 110. This process may alsobe referred to as keyword detection, with the wakeword being a specificexample of a keyword. Specifically, keyword detection is typicallyperformed without performing linguistic analysis, textual analysis orsemantic analysis. Instead, incoming audio (or audio data) is analyzedto determine if specific characteristics of the audio matchpreconfigured acoustic waveforms, audio signatures, or other data todetermine if the incoming audio “matches” stored audio datacorresponding to a keyword.

Thus, the wakeword detection module 220 may compare audio data to storedmodels or data to detect a wakeword. One approach for wakeword detectionapplies general large vocabulary continuous speech recognition (LVCSR)systems to decode the audio signals, with wakeword searching conductedin the resulting lattices or confusion networks. LVCSR decoding mayrequire relatively high computational resources. Another approach forwakeword spotting builds hidden Markov models (HMM) for each keywakeword word and non-wakeword speech signals respectively. Thenon-wakeword speech includes other spoken words, background noise etc.There can be one or more HMMs built to model the non-wakeword speechcharacteristics, which are named filler models. Viterbi decoding is usedto search the best path in the decoding graph, and the decoding outputis further processed to make the decision on keyword presence. Thisapproach can be extended to include discriminative information byincorporating hybrid DNN-HMM decoding framework. In another embodimentthe wakeword spotting system may be built on deep neural network(DNN)/recursive neural network (RNN) structures directly, without HMMinvolved. Such a system may estimate the posteriors of wakewords withcontext information, either by stacking frames within a context windowfor DNN, or using RNN. Following-on posterior threshold tuning orsmoothing is applied for decision making. Other techniques for wakeworddetection, such as those known in the art, may also be used.

Once the wakeword is detected, the user device 110 may “wake” and begintransmitting audio data 111 corresponding to input audio 11 to theserver(s) 120 for speech processing. Audio data corresponding to thataudio may be sent to a server 120 for routing to a recipient device ormay be sent to the server for speech processing for interpretation ofthe included speech (either for purposes of enablingvoice-communications and/or for purposes of executing a command in thespeech). The audio data 111 may include data corresponding to thewakeword, or the portion of the audio data corresponding to the wakewordmay be removed by the user device 110 prior to sending. Further, a userdevice 110 may “wake” upon detection of speech/spoken audio above athreshold, as described herein. Upon receipt by the server(s) 120, anASR module 250 may convert the audio data 111 into text. The ASRtranscribes audio data into text data representing the words of thespeech contained in the audio data. The text data may then be used byother components for various purposes, such as executing systemcommands, inputting data, etc. A spoken utterance in the audio data isinput to a processor configured to perform ASR which then interprets theutterance based on the similarity between the utterance andpre-established language models 254 stored in an ASR model storage 252c. For example, the ASR process may compare the input audio data withmodels for sounds (e.g., subword units or phonemes) and sequences ofsounds to identify words that match the sequence of sounds spoken in theutterance of the audio data.

The different ways a spoken utterance may be interpreted (i.e., thedifferent hypotheses) may each be assigned a probability or a confidencescore representing the likelihood that a particular set of words matchesthose spoken in the utterance. The confidence score may be based on anumber of factors including, for example, the similarity of the sound inthe utterance to models for language sounds (e.g., an acoustic model 253stored in an ASR Models Storage 252), and the likelihood that aparticular word which matches the sounds would be included in thesentence at the specific location (e.g., using a language or grammarmodel). Thus each potential textual interpretation of the spokenutterance (hypothesis) is associated with a confidence score. Based onthe considered factors and the assigned confidence score, the ASRprocess 250 outputs the most likely text recognized in the audio data.The ASR process may also output multiple hypotheses in the form of alattice or an N-best list with each hypothesis corresponding to aconfidence score or other score (such as probability scores, etc.).

The device or devices performing the ASR processing may include anacoustic front end (AFE) 256 and a speech recognition engine 258. Theacoustic front end (AFE) 256 transforms the audio data from themicrophone into data for processing by the speech recognition engine.The speech recognition engine 258 compares the speech recognition datawith acoustic models 253, language models 254, and other data models andinformation for recognizing the speech conveyed in the audio data. TheAFE may reduce noise in the audio data and divide the digitized audiodata into frames representing a time intervals for which the AFEdetermines a number of values, called features, representing thequalities of the audio data, along with a set of those values, called afeature vector, representing the features/qualities of the audio datawithin the frame. Many different features may be determined, as known inthe art, and each feature represents some quality of the audio that maybe useful for ASR processing. A number of approaches may be used by theAFE to process the audio data, such as mel-frequency cepstralcoefficients (MFCCs), perceptual linear predictive (PLP) techniques,neural network feature vector techniques, linear discriminant analysis,semi-tied covariance matrices, or other approaches known to those ofskill in the art.

The speech recognition engine 258 may process the output from the AFE256 with reference to information stored in speech/model storage (252).Alternatively, post front-end processed data (such as feature vectors)may be received by the device executing ASR processing from anothersource besides the internal AFE. For example, the device 110 may processaudio data into feature vectors (for example using an on-device AFE 256)and transmit that information to a server across a network 199 for ASRprocessing. Feature vectors may arrive at the server encoded, in whichcase they may be decoded prior to processing by the processor executingthe speech recognition engine 258.

The speech recognition engine 258 attempts to match received featurevectors to language phonemes and words as known in the stored acousticmodels 253 and language models 254. The speech recognition engine 258computes recognition scores for the feature vectors based on acousticinformation and language information. The acoustic information is usedto calculate an acoustic score representing a likelihood that theintended sound represented by a group of feature vectors matches alanguage phoneme. The language information is used to adjust theacoustic score by considering what sounds and/or words are used incontext with each other, thereby improving the likelihood that the ASRprocess will output speech results that make sense grammatically. Thespecific models used may be general models or may be modelscorresponding to a particular domain, such as music, banking, etc.

The speech recognition engine 258 may use a number of techniques tomatch feature vectors to phonemes, for example using Hidden MarkovModels (HMMs) to determine probabilities that feature vectors may matchphonemes. Sounds received may be represented as paths between states ofthe HMM and multiple paths may represent multiple possible text matchesfor the same sound.

Following ASR processing, the ASR results may be sent by the speechrecognition engine 258 to other processing components, which may belocal to the device performing ASR and/or distributed across thenetwork(s) 199. For example, ASR results in the form of a single textualrepresentation of the speech, an N-best list including multiplehypotheses and respective scores, lattice, etc. may be sent to a server,such as server 120, for natural language understanding (NLU) processing,such as conversion of the text into commands for execution, either bythe device 110, by the server 120, or by another device (such as aserver running a specific application like a search engine, etc.).

The device performing NLU processing 260 (e.g., server 120) may includevarious components, including potentially dedicated processor(s),memory, storage, etc. A device configured for NLU processing may includea named entity recognition (NER) module 252 and intent classification(IC) module 264, a result ranking and distribution module 266, and NLUstorage 273. The NLU process may also utilize gazetteer information (284a-284 n) stored in entity library storage 282. The gazetteer informationmay be used for entity resolution, for example matching ASR results withdifferent entities (such as song titles, contact names, etc.) Gazetteersmay be linked to users (for example a particular gazetteer may beassociated with a specific user's music collection), may be linked tocertain domains (such as shopping), or may be organized in a variety ofother ways.

The NLU process takes textual input (such as processed from ASR 250based on the utterance 11) and attempts to make a semanticinterpretation of the text. That is, the NLU process determines themeaning behind the text based on the individual words and thenimplements that meaning. NLU processing 260 interprets a text string toderive an intent or a desired action from the user as well as thepertinent pieces of information in the text that allow a device (e.g.,device 110) to complete that action. For example, if a spoken utteranceis processed using ASR 250 and outputs the text “call mom” the NLUprocess may determine that the user intended to activate a telephone inhis/her device and to initiate a call with a contact matching the entity“mom.”

The NLU may process several textual inputs related to the sameutterance. For example, if the ASR 250 outputs N text segments (as partof an N-best list), the NLU may process all N outputs to obtain NLUresults.

The NLU process may be configured to parse, tag, and annotate text aspart of NLU processing. For example, for the text “call mom,” “call” maybe tagged as a command (to execute a phone call) and “mom” may be taggedas a specific entity and target of the command (and the telephone numberfor the entity corresponding to “mom” stored in a contact list may beincluded in the annotated result).

To correctly perform NLU processing of speech input, the NLU process 260may be configured to determine a “domain” of the utterance so as todetermine and narrow down which services offered by the endpoint device(e.g., server 120 or device 110) may be relevant. For example, anendpoint device may offer services relating to interactions with atelephone service, a contact list service, a calendar/schedulingservice, a music player service, etc. Words in a single text query mayimplicate more than one service, and some services may be functionallylinked (e.g., both a telephone service and a calendar service mayutilize data from the contact list).

The name entity recognition module 262 receives a query in the form ofASR results and attempts to identify relevant grammars and lexicalinformation that may be used to construe meaning. To do so, a nameentity recognition module 262 may begin by identifying potential domainsthat may relate to the received query. The NLU storage 273 includes adatabases of devices (274 a-274 n) identifying domains associated withspecific devices. For example, the device 110 may be associated withdomains for music, telephony, calendaring, contact lists, anddevice-specific communications, but not video. In addition, the entitylibrary may include database entries about specific services on aspecific device, either indexed by Device ID, User ID, or Household ID,or some other indicator.

A domain may represent a discrete set of activities having a commontheme, such as “shopping”, “music”, “calendaring”, etc. As such, eachdomain may be associated with a particular language model and/or grammardatabase (276 a-276 n), a particular set of intents/actions (278 a-278n), and a particular personalized lexicon (286). Each gazetteer (284a-284 n) may include domain-indexed lexical information associated witha particular user and/or device. For example, the Gazetteer A (284 a)includes domain-index lexical information 286 aa to 286 an. A user'smusic-domain lexical information might include album titles, artistnames, and song names, for example, whereas a user's contact-listlexical information might include the names of contacts. Since everyuser's music collection and contact list is presumably different, thispersonalized information improves entity resolution.

A query is processed applying the rules, models, and informationapplicable to each identified domain. For example, if a querypotentially implicates both communications and music, the query will beNLU processed using the grammar models and lexical information forcommunications, and will be processed using the grammar models andlexical information for music. The responses based on the query producedby each set of models is scored (discussed further below), with theoverall highest ranked result from all applied domains is ordinarilyselected to be the correct result.

An intent classification (IC) module 264 parses the query to determinean intent or intents for each identified domain, where the intentcorresponds to the action to be performed that is responsive to thequery. Each domain is associated with a database (278 a-278 n) of wordslinked to intents. For example, a music intent database may link wordsand phrases such as “quiet,” “volume off,” and “mute” to a “mute”intent. The IC module 264 identifies potential intents for eachidentified domain by comparing words in the query to the words andphrases in the intents database 278.

In order to generate a particular interpreted response, the NER 262applies the grammar models and lexical information associated with therespective domain. Each grammar model 276 includes the names of entities(i.e., nouns) commonly found in speech about the particular domain(i.e., generic terms), whereas the lexical information 286 from thegazetteer 284 is personalized to the user(s) and/or the device. Forinstance, a grammar model associated with the shopping domain mayinclude a database of words commonly used when people discuss shopping.

The intents identified by the IC module 264 are linked todomain-specific grammar frameworks (included in 276) with “slots” or“fields” to be filled. For example, if “play music” is an identifiedintent, a grammar (276) framework or frameworks may correspond tosentence structures such as “Play {Artist Name},” “Play {Album Name},”“Play {Song name},” “Play {Song name} by {Artist Name},” etc. However,to make recognition more flexible, these frameworks would ordinarily notbe structured as sentences, but rather based on associating slots withgrammatical tags.

For example, the NER module 260 may parse the query to identify words assubject, object, verb, preposition, etc., based on grammar rules andmodels, prior to recognizing named entities. The identified verb may beused by the IC module 264 to identify intent, which is then used by theNER module 262 to identify frameworks. A framework for an intent of“play” may specify a list of slots/fields applicable to play theidentified “object” and any object modifier (e.g., a prepositionalphrase), such as {Artist Name}, {Album Name}, {Song name}, etc. The NERmodule 260 then searches the corresponding fields in the domain-specificand personalized lexicon(s), attempting to match words and phrases inthe query tagged as a grammatical object or object modifier with thoseidentified in the database(s).

This process includes semantic tagging, which is the labeling of a wordor combination of words according to their type/semantic meaning.Parsing may be performed using heuristic grammar rules, or an NER modelmay be constructed using techniques such as hidden Markov models,maximum entropy models, log linear models, conditional random fields(CRF), and the like.

For instance, a query of “play mother's little helper by the rollingstones” might be parsed and tagged as {Verb}: “Play,” {Object}:“mother's little helper,” {Object Preposition}: “by,” and {ObjectModifier}: “the rolling stones.” At this point in the process, “Play” isidentified as a verb based on a word database associated with the musicdomain, which the IC module 264 will determine corresponds to the “playmusic” intent. No determination has been made as to the meaning of“mother's little helper” and “the rolling stones,” but based on grammarrules and models, it is determined that these phrase relate to thegrammatical object of the query.

The frameworks linked to the intent are then used to determine whatdatabase fields should be searched to determine the meaning of thesephrases, such as searching a user's gazette for similarity with theframework slots. So a framework for “play music intent” might indicateto attempt to resolve the identified object based {Artist Name}, {AlbumName}, and {Song name}, and another framework for the same intent mightindicate to attempt to resolve the object modifier based on {ArtistName}, and resolve the object based on {Album Name} and {Song Name}linked to the identified {Artist Name}. If the search of the gazetteerdoes not resolve the slot/field using gazetteer information, the NERmodule 262 may search the database of generic words associated with thedomain (in the storage 273). So for instance, if the query was “playsongs by the rolling stones,” after failing to determine an album nameor song name called “songs” by “the rolling stones,” the NER 262 maysearch the domain vocabulary for the word “songs.” In the alternative,generic words may be checked before the gazetteer information, or bothmay be tried, potentially producing two different results.

The comparison process used by the NER module 262 may classify (i.e.,score) how closely a database entry compares to a tagged query word orphrase, how closely the grammatical structure of the query correspondsto the applied grammatical framework, and based on whether the databaseindicates a relationship between an entry and information identified tofill other slots of the framework.

The NER modules 262 may also use contextual operational rules to fillslots. For example, if a user had previously requested to pause aparticular song and thereafter requested that the voice-controlleddevice to “please un-pause my music,” the NER module 262 may apply aninference-based rule to fill a slot associated with the name of the songthat the user currently wishes to play—namely the song that was playingat the time that the user requested to pause the music.

The results of NLU processing may be tagged to attribute meaning to thequery. So, for instance, “play mother's little helper by the rollingstones” might produce a result of: {domain} Music, {intent} Play Music,{artist name} “rolling stones,” {media type} SONG, and {song title}“mother's little helper.” As another example, “play songs by the rollingstones” might produce: {domain} Music, {intent} Play Music, {artistname} “rolling stones,” and {media type} SONG.

The output from the NLU processing (which may include tagged text,commands, etc.) may then be sent to a command processor 290, which maybe located on a same or separate server 120 as part of system 100. Thedestination command processor 290 may be determined based on the NLUoutput. For example, if the NLU output includes a command to play music,the destination command processor 290 may be a music playingapplication, such as one located on device 110 or in a music playingappliance, configured to execute a music playing command. If the NLUoutput includes a search request, the destination command processor 290may include a search engine processor, such as one located on a searchserver, configured to execute a search command.

The NLU operations of the system described herein may take the form of amulti-domain architecture, such as that illustrated in FIG. 3. In themulti-domain architecture, each domain (which may include a set ofintents and entity slots that define a larger concept such as music,books etc.) is constructed separately and made available to the NLUcomponent 260 during runtime operations where NLU operations areperformed on text (such as text output from the ASR component 250). Eachdomain may have specially configured components to perform various stepsof the NLU operations. For example, a message domain 302 (Domain A) mayhave an NER component 262-A that identifies what slots (i.e., portionsof input text) may correspond to particular entities relevant to thatdomain. The NER component 262-A may use a machine learning model, suchas a domain specific conditional random field (CRF) to both identify theportions corresponding to an entity as well as identify what type ofentity corresponds to the text portion. For example, for the text “tellJohn Smith I said hello,” an NER 262-A trained for the message domain302 may recognize the portion of text [John Smith] corresponds to anentity. The message domain 302 may also have its own intentclassification (IC) component 264-A that determines the intent of thetext assuming that the text is within the proscribed domain. The ICcomponent may use a model, such as a domain specific maximum entropyclassifier to identify the intent of the text. The message domain 302may also have its own slot filling component 310-A that can apply rulesor other instructions to standardize labels or tokens from previousstages into an intent/slot representation. The precise transformationmay depend on the domain (for example, for a travel domain a textmention of “Boston airport” may be transformed to the standard BOSthree-letter code referring to the airport). The message domain 302 mayalso have its own entity resolution component 312-A that can refer to anauthority source (such as a domain specific knowledge base) that is usedto specifically identify the precise entity referred to in the entitymention identified in the incoming text. Specific intent/slotcombinations may also be tied to a particular source, which may then beused to resolve the text (such as by providing information or a commandto be executed in response to a user query). The output from the entityresolution component 312-A may include a command, information, or otherNLU result data indicating how the domain specific NLU processinghandled the text and how the system should respond to the text,according to that specific domain.

As illustrated in FIG. 3, multiple domains may operate substantially inparallel, with different domain specific components. Moreover, eachdomain may implement certain protocols when exchanging messages or othercommunications. That is, domain B, for real-time calls, 304 may have itsown NER component 262-B, IC module 264-B, slot filling component 310-B,and entity resolution component 312-B. The system may includingadditional domains not herein described. The same text that is inputinto the NLU pipeline for domain A 302 may also be input into the NLUpipeline for domain B 304, where the components for domain B 304 willoperate on the text as if the text related to domain B, and so on forthe different NLU pipelines for the different domains. Each domainspecific NLU pipeline will create its own domain specific NLU results,for example NLU results A (for domain A), NLU results B (for domain B),NLU results C (for domain C), and so on.

Such a multi-domain architecture results in narrowly defined intents andslots that are particular for each specific domain. This is due, inpart, to the different models and components (such as the domainspecific NER component, IC module, etc. and related models) beingtrained to operate only for the designated domain. Further, theseparation of domains results in similar actions being representedseparately across the domains even if there is overlap in the action.For example, “next song,” “next book,” and “next” may all be indicatorsof the same action, but will be defined differently in different domainsdue to domain specific processing restrictions.

The server 120 may also include data regarding user accounts, shown bythe user profile storage 402 illustrated in FIG. 4. The user profilestorage may be located proximate to server 120, or may otherwise be incommunication with various components, for example over network 199. Theuser profile storage 402 may include a variety of information related toindividual users, accounts, etc. that interact with the system 100. Forillustration, as shown in FIG. 4, the user profile storage 402 mayinclude data regarding the devices associated with particular individualuser accounts 404. In an example, the user profile storage 402 is acloud-based storage. Such data may include device identifier (ID) andinternet protocol (IP) address information for different devices as wellas names of users and locations of the devices. The user profile storagemay additionally include communication alteration triggers specific toeach device, indication preferences for each device, etc. In an example,the type of indication to be output by each device may not be stored ina user profile. Rather, the type of indication may be dependent uponcontext. For example, if video messages are being exchanged by thesystem, the indication may be visual. For further example, if audiomessages are being exchanged by the system, the indication may beaudible.

Each user profile may store one or more communication alteration paths.Moreover, each communication alteration path may include a singlecommunication alteration trigger or multiple communication alterationtriggers that represent when communication alteration should occur. Itshould be appreciated that N number of communication alteration pathshaving M number of communication alteration triggers may be stored in asingle user profile. Each communication alteration path may be unique toa different individual with which the user communicates. For example,one communication alteration path may be used when the user communicateswith its mom, another communication alteration path may be used when theuser communicates with its spouse, etc. Each communication alterationpath may also be unique to a type of communication (e.g., audiomessaging, video messaging, etc.). Each communication alteration pathmay also be unique to the type of device(s) involved in thecommunication. For example, a user may have a first communicationalteration path configured for a device in the user's car, a secondcommunication alteration path configured for a device in the user'sbedroom, etc.

Some or all of the communication alteration paths of a user profile maybe dynamic. That is the communication alteration paths may depend uponexternal signals. An illustrative external signal includes proximity toa device. For example, one communication alteration path may be usedwhen communicating with the user's mom while the user's mom is notproximate to her device, and a second communication alteration path maybe used when communicating with the user's mom while the user's mom isproximate to her device. For example, a speech-controlled device 110 maycapture one or more images, and send image data corresponding thereto tothe server 120. The server 120 may determine the image data includes arepresentation of a human. The server 120, may also determine aproximity of the human to the device 110 based on a location of therepresentation of the human in the image data. Dynamic choosing ofcommunication alteration paths may also be influenced by machinelearning. For example, a communication alteration path may be configuredto alter communications to real-time calls when the user iscommunicating with its mom after a certain time at night. The system maythen determine that a certain percentage of the time, the user altersthe communication within a threshold amount of time. Based on thisdetermination, the system may suggest the user revise/update thecommunication alteration path to not alter messaging to real-time callsso quickly.

Each communication escalation path may include one or more communicationalterations. One type of communication alteration involves removing theneed for a wakeword portion so spoken audio only needs to include acommand (e.g., language causing the system to send a message) andmessage content. A second type of communication alteration involvesremoving the need for a wakeword portion and a command so spoken audioonly needs to include message content. A third type of communicationalteration involves replacing a default wakeword, and make the wakewordthe name of the recipient of the message (e.g., mom, John, etc.). Afourth type of communication alteration is altering a message exchangeto a real-time call.

FIGS. 5A through 5D illustrate the alteration of voice-basedinteractions via speech-controlled devices. A first speech-controlleddevice 110 a captures spoken audio including a wakeword portion and apayload portion (illustrated as 502). For example, the speech-controlleddevice 110 a may be in a sleep mode until detection of a spokenwakeword, which triggers the speech-controlled device 110 a to wake andcapture audio (which may include the spoken wakeword and speechthereafter) for processing and sending to the server 120. Thespeech-controlled device 110 a sends audio data corresponding to thecaptured spoken audio to the server 120 (illustrated as 504).

The server 120 performs ASR on the received audio data to determine text(illustrated as 506). The server 120 may determine the wakeword portionand the payload portion of the text, and perform NLU on the payloadportion (illustrated as 508). Performing NLU processing may include theserver 120 tagging recipient information of the payload portion(illustrated as 510), tagging message content information of the payloadportion (illustrated as 512), and tagging the overall payload portionwith a “send message” intent tag (illustrated as 514). For example, thepayload portion of the received audio data may correspond to text of“tell John Smith I said hello.” According to this example, the server120 may tag “John Smith” as recipient information, may tag “hello” asmessage content information, and may tag the utterance with the “sendmessage” intent tag. Tagging the payload portion with the message intenttag may be performed using the message domain 302 and/or may cause thesystem to perform further messaging commands, such as with a messagingcommand processor 290.

Using the tagged recipient information, the server 120 determines adevice associated with the recipient (e.g., the speech-controlled device110 b) (illustrated as 516 in FIG. 5B). To determine the recipientdevice, the server 120 may use a user profile associated with thespeech-controlled device 110 a and/or the user that spoke the initialaudio. For example, the server 120 may access a table of the userprofile to match text in the table corresponding to the tagged recipientinformation (i.e., “John Smith”). Once matching text is identified, theserver 120 may identify the recipient device associated with thematching text in the table.

The server 120 also generates output audio data using a domain andassociated protocol(s) of the server 120 associated with the “sendmessage” intent tag (illustrated as 518). The output audio data mayinclude the spoken audio received from the speech-controlled device 110a. Alternatively, the output audio data may include computer-generated,text-to-speech (TTS) audio data based on the text of the message contentreceived from the speech-controlled device 110 a. The server 120 sendsthe output audio data to the recipient device (illustrated as 520),which outputs audio data to the recipient (illustrated as 522). In anexample, the speech-controlled device 110 b of the recipient may notoutput the audio data until it detects a command from the recipient todo so. Such a command may be a spoken utterance of the recipientcorresponding to “What are my messages?”, “Do I have any messages?”,etc.

The server 120 performs message communications between the firstspeech-controlled device 110 a and the second speech-controlled device110 b as detailed herein above with respect to steps 502-522 of FIGS. 5Aand 5B (e.g., via the message domain) (illustrated as 524), until theserver 120 determines the occurrence of a communication alterationtrigger (illustrated as 526). A communication alteration trigger maycause the server 120 to perform subsequent communications/processesusing another domain and corresponding protocol(s), different from thedomain used to perform previous communications/processes. Alternatively,the system may adjust the processing of future messages to not requirecertain spoken data (such as a wakeword or indication of a recipient).The determined communication alteration trigger may take on many forms.The communication alteration trigger may be based on whether a thresholdnumber of message exchanges between the first speech-controlled device110 a and the second speech-controlled device 110 b is met or exceeded.For example, the threshold number of message exchanges may be set by auser of either of the speech-controlled devices 110 a/110 b, and may berepresented in a respective user profile. It should be appreciated thatthe threshold number of message exchanges associated with a user profileof the first speech-controlled device 110 a may be different from thethreshold number of message exchanges associated with a user profile ofthe second speech-controlled device 110 b. In this instance, thethreshold used by the server 120 to determine when communicationalteration should occur may be the threshold that is met or exceededfirst (i.e., the threshold having a less number of required messageexchanges). The communication alteration trigger may also oralternatively be based on a threshold number of message exchangesoccurring with a threshold amount of time. For example, the thresholdnumber of message exchanges and/or the threshold amount of time may beset by a user of either of the speech-controlled devices 110 a/110 b,and may be represented in a respective user profile. It should beappreciated that the threshold number of message exchanges and thethreshold amount of time associated with a user profile of the firstspeech-controlled device 110 a may be different from the thresholdnumber of message exchanges associated with a user profile of the secondspeech-controlled device 110 b. In this instance, the thresholds used bythe server 120 to determine when communication alteration should occurmay be the threshold that is met or exceeded first. The communicationalteration trigger may also or alternatively be based on users of bothof the speech-controlled devices 110 a/110 b simultaneously being withthreshold distances of their respective device. It should be appreciatedthat communication alteration may occur based on the satisfaction of asingle communication alteration trigger. It should also be appreciatedthat communication alteration may occur based on satisfaction of morethan one communication alteration trigger.

Once one or more communication alteration triggers are determined,depending upon implementation, the server 120 reconfigures utterancesfrom the first/second speech-controlled device to not require presenceof a wakeword portion or recipient information in received audio data(illustrated as 528). This may be done using the message domain 302 andassociated protocol(s), for example. In addition, the reconfigurationthat occurs at step 528 may instruct the speech-controlled device 110 bto output a received communication without first detecting speechcorresponding to a command to do so. Further, the server 120 may send asignal to one or both of the speech-controlled devices 110 a/110 bindicating the communication between the first and secondspeech-controlled devices 110 a/110 b is being altered (illustrated as530). A speech-controlled device may output an indication representingthe device is “listening” in an attempt to capture message content. Inaddition, a speech-controlled device may output an indicationrepresenting a recipient's device is capturing spoken message content.The speech-controlled device 110 a and/or the speech-controlled device110 b may then output a signal representing that wakeword audio is nolonger required (illustrated as 532 in FIG. 5C). The signal output byone or both of the speech-controlled devices 110 a/110 b may be a staticindication or motion indication as described herein below.

Thereafter, the speech-controlled device 110 a captures spoken audiofrom a user including only payload information (illustrated as 534), andsends audio data corresponding to the payload information to the server120 (illustrated as 536). The server 120 performs ASR on the receivedaudio data to determine text (illustrated as 538), and performs NLUprocessing on the payload information text (illustrated as 540).Performing NLU processing may include the server 120 tagging recipientinformation of the payload information text, tagging message contentinformation of the payload information text, and tagging the overallpayload information text with an instant message intent tag. Forexample, the payload information of the received audio data may state“When will you be done with the project?” According to this example, theserver 120 may tag “when will you be done with the project” as messagecontent information, and may tag the utterance with a “send instantmessage” intent tag. Tagging the payload information text with themessage intent tag may cause the server 120 to perform downstreamprocesses using the message domain 302. By not requiring recipientinformation to be present in the input audio, the server 120 may assumethe recipient device is the same as the recipient device used inprevious communications, thereby negating the need of the server 120 toagain determine the recipient device.

The server 120 generates output audio data using a domain and associatedprotocol(s) of the server 120 associated with the “send instant message”intent tag (illustrated as 542). For example, the message domain 302 maybe associated with the instant message intent tag. The output audio datamay include the spoken audio received from the speech-controlled device110 a. Alternatively, the output audio data may includecomputer-generated, text-to-speech (TTS) audio data based on the spokenaudio received from the speech-controlled device 110 a. The server 120sends the output audio data to the recipient device (i.e., thespeech-controlled device 110 b) (illustrated as 544), which outputsaudio of the audio data to the recipient (illustrated as 546 in FIG.5D). As detailed above, the reconfiguration that occurs at step 528 mayinstruct speech-controlled device 110 b to output a receivedcommunication without first receiving a command from a user to do so. Assuch, it should be appreciated that the speech-controlled device 110 bmay output the audio data to the recipient at step 546 without firstreceiving a command to do so. That is, the speech-controlled device 110b may auto-play the audio data.

The server 120 performs instant message communications between the firstspeech-controlled device 110 a and the second speech-controlled device110 b as detailed herein above with respect to steps 534-546 of FIGS. 5Cthrough 5D (e.g., via the instant message domain and without requiringwakeword audio data) (illustrated as 548), until the server 120determines the occurrence of another communication alteration trigger(illustrated as 550). The second determined communication alterationtrigger may take on many forms. Like the first communication alterationtrigger, the second communication alteration trigger may be based onwhether a threshold number of message exchanges between the firstspeech-controlled device 110 a and the second speech-controlled device110 b is met or exceeded, based on a threshold number of messageexchanges occurring with a threshold amount of time, and/or based onusers of both of the speech-controlled devices 110 a/110 bsimultaneously being with threshold distances of their respectivedevice. The thresholds used in determining the first and secondcommunication alteration triggers may be the same (e.g., each requires 5message exchanges) or different (e.g., the first communicationalteration occurs after 5 message exchanges using the message domain 302and the second communication alteration occurs after 7 message exchangesusing the message domain 302). The message exchanges for eachcommunication alteration trigger may be determined using a singlecounter that does not reset after the first communication alteration.According to the previous example, the first communication alterationmay occur after the counter reaches 5 message exchanges (i.e., 5 messageexchanges using the message domain 302) and the second communicationalteration may occur after the counter reaches 12 message exchanges(i.e., 7 message exchanges using the message domain 302). Alternatively,the message exchanges for each communication alteration may bedetermined using different counters, or a single counter that resetsafter the first communication alteration. According to the previousexample, the first communication alteration may occur after the counterreaches 5 message exchanges (i.e., 5 message exchanges using the messagedomain 302), the counter may then reset to zero, and the secondcommunication alteration may occur after the counter reaches 7 messageexchanges (i.e., 7 message exchanges using the message domain 302). Thethreshold distances to the speech-controlled devices 110 a/110 b withinwhich users need to be for the first and second communicationalterations may be the same or different. Moreover, like the firstcommunication alteration, the second communication alteration may occurbased on satisfaction of a single communication alteration trigger, ormore than one communication alteration trigger.

Once the second communication alteration trigger(s) is determined,depending upon implementation, the server 120 reconfigures to use adomain and associated protocol(s) that establishes a real-time callbetween the speech-controlled device 110 a and the speech-controlleddevice 110 b (illustrated as 552). Such a domain may be the real-timecall domain 304, for example. A real-time call, as used herein, refersto a call that is facilitated between the speech-controlled devices 110a/110 b via the server 120, where a direct communication channel may beopened between the speech controlled devices. For example, during areal-time call, the system may send audio data from the firstspeech-controlled device 110 a to the second speech-controlled device110 b without performing speech processing (such as ASR or NLU) on theaudio data, thus enabling the user of the first speech-controlled device110 a to “speak directly” with the user of the second speech-controlleddevice 110 b. Alternatively, the system may perform speech processing(such as ASR or NLU) but absent a command intended for the system, maypass the audio data back and forth between the devices 110 a/110 b. Areal-time call can be ended, for example, as discussed below inreference to FIG. 7.

The server 120 may send a signal to one or both of the speech-controlleddevices 110 a/110 b indicating a real-time call is established(illustrated as 554). The speech-controlled device 110 a and/or thespeech-controlled device 110 b then outputs a signal representing theuser can speak as if s/he were conducting a point-to-point call(illustrated as 556). A real-time or point-to-point call/communication,as used herein, refers to a call that is facilitated between thespeech-controlled devices 110 a/110 b via the server 120. That is, areal-time call or point-to-point call is a communication where audio issimply captured by a device, sent as audio data to the server, and theserver merely sends the received audio data to a recipient device, withthe recipient device outputting audio without first receiving a commandto do so. The signal output by one or both of the speech-controlleddevices 110 a/110 b may be a static indication or motion indication asdescribed herein below. The system then performs the real-timecommunication session (illustrated as 558). The real-time communicationsession may be performed by the system until a de-escalation trigger (asdetailed herein) is determined.

When performing communications between the speech-controlled devices,the system may use various types of protocols that control data size,transmission speed, etc. For example, a first protocol may be used tocontrol the exchange of communications that require the presence of awakeword portion and recipient content. A second protocol may be used tocontrol the exchange of communications that do not require a wakewordportion, but still require recipient content. A third protocol may beused to control the exchange of communications that do not contain NLUintent. That is, the third protocol may be used when neither a wakewordportion nor recipient content is required, as the system presumes therecipient based on past contemporaneous message exchanges. A real-timeprotocol, such as a VoIP, may be used when a synchronous call betweenspeech-controlled devices is performed.

FIGS. 6A and 6B illustrate alteration of voice-based interactions viaspeech-controlled devices based on the intended recipient of themessage. The first speech-controlled device 110 a captures spoken audioincluding a wakeword portion and a payload portion (illustrated as 502).For example, the speech-controlled device 110 a may be in a sleep modeuntil detection of a spoken wakeword, which triggers thespeech-controlled device 110 a to wake and capture audio including thespoken wakeword and speech thereafter. The speech-controlled device 110a sends audio data corresponding to the captured spoken audio to theserver 120 (illustrated as 504).

The server 120 performs ASR on the received audio data to determine text(illustrated as 506). The server 120 determines the wakeword portion andthe payload portion of the text, and performs NLU on the payload portion(illustrated as 508). Performing NLU processing may include the server120 tagging recipient information of the payload portion (illustrated as510), tagging message content information of the payload portion(illustrated as 512), and tagging the overall payload portion with a“send message” intent tag (illustrated as 514). For example, the payloadportion of the received audio data may state “tell mom I said I will bethere soon.” According to this example, the server 120 may tag “mom” asrecipient information, may tag “I will be there soon” as message contentinformation, and may associate the utterance with the “send message”intent tag. As detailed herein above, communication alteration paths andcommunication alteration triggers may be configurable via user profiles.According to this embodiment, the server 120 may determine communicationalteration based on the intended recipient of the message. For example,the server 120, using the tagged recipient information, may access auser profile of the speech-controlled device 110 a and determine acommunication alteration path that indicates communications with “mom”are to be performed via real-time calls (illustrated as 602 in FIG. 6B).Thereafter, the server 120 reconfigures to use a domain and associatedprotocol(s) that establishes a real-time call between thespeech-controlled device 110 a and the speech-controlled device 110 b(illustrated as 552). Such a domain may be the real-time call domain304, for example. The server 120 may send a signal to one or both of thespeech-controlled devices 110 a/110 b indicating a real-time call isestablished (illustrated as 554). The speech-controlled device 110 aand/or the speech-controlled device 110 b then outputs a signalrepresenting a user can speak as if s/he were conducting apoint-to-point call (illustrated as 556). The signal output by one orboth of the speech-controlled devices 110 a/110 b may be a staticindication or motion indication as described herein below. The systemthen performs the real-time communication session (illustrated as 558).The real-time communication session may be performed by the system untilanother communication alteration trigger (as detailed herein) isdetermined.

FIG. 7 illustrates alteration of a voice-based interaction viaspeech-controlled devices. The server 120 exchanges communicationsbetween the speech-controlled devices 110 a/110 b via a domain andassociated protocol(s) associated with real-time calls (illustrated as702) until the server 120 determines the occurrence of a communicationalteration trigger (illustrated as 704). Such a domain may be thereal-time call domain 304, for example. The communication alterationtrigger may take on various forms. The communication alteration triggermay be based on users of either of the speech-controlled devices 110a/110 b multitasking (i.e., causing the server 120 to perform tasks notassociated with a real-time call). The communication alteration triggermay also or alternatively be based on a threshold period of inactivitybeing met or exceeded (e.g., a determination that an exchange has nothappened with n amounts of minutes). The communication alterationtrigger may also or alternatively be based on a user directive (e.g., auser of either of the speech-controlled devices 110 a/110 b stating, forexample, “close the call,” “stop the call,” “cease the call,” etc.). Thecommunication alteration trigger may also or alternatively be based onindications originating from users of both the speech-controlled devices110 a/110 b (e.g., both the users stating “bye,” “goodbye,” etc. withina threshold period of seconds of each other). In addition, thecommunication alteration trigger may also or alternatively be based onthe server 120 detecting a wakeword in an exchange of the real-timecall. Communication alteration may occur based on satisfaction of one ormore than one communication alteration triggers being determined.

After determining alteration should occur, the server 120 ceases thereal-time call (illustrated as 706) and sends a signal representing suchto one or both of the speech-controlled devices 110 a/110 b (illustratedas 708). The speech-controlled device 110 a and/or the speech-controlleddevice 110 b then outputs a signal representing the real-time call hasceased (illustrated as 710). The signal output by one or both of thespeech-controlled devices 110 a/110 b may be a static indication ormotion indication as described herein below. Altering the communicationmay involve ceasing all communication between the speech-controlleddevices 110 a/110 b at that point in time. Alternatively, altering thecommunication may involve changing the communication to a second form,different from a real-time call. For example, the second form ofcommunication may involve the server 120 performing instant messagecommunications between the first speech-controlled device 110 a and thesecond speech-controlled device 110 b as detailed herein above withrespect to steps 534-546 of FIGS. 5C through 5D (illustrated as 548),until the server 120 determines the occurrence of a communicationalteration trigger.

FIGS. 8A and 8B illustrate the output of signaling via user interfacesof speech-controlled devices. The speech-controlled device 110 acaptures spoken audio (illustrated as 802), compiles the captured spokenaudio into audio data, and sends the audio data to the server 120(illustrated as 504).

The server 120 performs ASR on the audio data to determine text (e.g.,“tell John Smith I said hello”) (illustrated as 506) and performs NLU onthe text (illustrated as 804). The server 120 locates tagged recipientinformation (i.e., “John Smith”) within the NLU processed text(illustrated as 806) and therefrom determines a recipient device(illustrated as 808). For example, the server 120 may access a userprofile associated with the speech-controlled device 110 a and/or itsuser. Using the user profile, the server 120 may locate textcorresponding to the recipient information (i.e., “John Smith”) within atable, and may identify recipient device information associated with therecipient information within the table. The server 120 also determinestagged message content (e.g., “hello”) within the NLU processed text(illustrated as 810).

The server 120 sends a signal indicating the message content is or willbe sent to the recipient device (i.e., the speech-controlled device 110b) to the speech-controlled device 110 a from which the initial spokenaudio data originated (illustrated as 812). In response to receiving themessage, the speech-controlled device 110 a outputs a visual indicationrepresenting the message content (i.e., hello) is or will be sent to therecipient device (illustrated as 814). For example, a visual indicationmay include outputting a static indicator (e.g., a certain color, etc.)or a motion indicator (e.g., a blinking or strobing element, continuousmovement, etc.). The visual indication output may be configuredaccording to a user profile preference. Optionally, in response toreceiving the message, the speech-controlled device 110 may output atactile and/or an audible indication (illustrated as 816). The tactileindication may include the speech-controlled device 110 a vibratingand/or a remote device in communication with the speech-controlleddevice 110 a (e.g., a smart watch) vibrating. The remote device and thespeech-controlled device 110 a may be in communication by being locatedin a single table of user devices associated with a user profile. Theaudible indication may include computer generated/TTS generated speechand/or user generated speech corresponding to, for example, “yourmessage is being sent” or “your message will be sent momentarily.” Theaudible indication, like the tactile indication, may be output by thespeech-controlled device 110 a, a remote microphone array, and/or aremote device (e.g., a smart watch). The remote device, the microphonearray, and the speech-controlled device 110 a may be in communication bybeing located in a single table of user devices associated with a userprofile.

The server 120 also sends audio data including the message content tothe determined recipient device (i.e., the speech-controlled device 110b) (illustrated as 818). It should be appreciated that steps 814-818 (aswell as other steps of other figures) may occur in various orders, andmay also occur concurrently. The speech-controlled device 110 b thenoutputs audio corresponding to the message content (illustrated as 522).When the speech-controlled device 110 b detects speech responding to themessage content (illustrated as 820), and the speech-controlled device110 b sends a signal representing such to the server 120 (illustrated as822). The server 120 then sends a signal to the speech-controlled device110 a indicating the speech-controlled device 110 b is detecting speech(illustrated as 824). The server 120 may determine the detected speechis in response to the output audio based on, for example, the name ofthe recipient indicated in the detected speech or the speech-controlleddevices 110 a/110 b being part of an instant message exchange that doesnot require wakeword audio data. In addition, in an example, the server120 may cause the speech-controlled device 110 b to output audio askinga user whether the user want to reply to the received message. Theserver 120 may then receive audio data from the second speech-controlleddevice 110 b, perform ASR on the audio data to determine text data,determine the text data includes at least one word indicating an intentto respond (e.g., yes), and therefrom determines audio data receivedthereafter is in response to the original message. In another example,the server 120 may receive audio data from the second speech-controlleddevice 110 b, determine using speech processing that an audio signatureof the received audio data matches a voice-based speaker ID of therecipient of the original message, and therefrom determine audio datareceived thereafter is in response to the original message. In responseto receiving the signal, the speech-controlled device 110 a outputs avisual indication representing the speech-controlled device 110 b isdetecting speech (illustrated as 826). For example, the visualindication may include outputting a static indicator (e.g., a certaincolor, etc.) or a motion indicator (e.g., a blinking or strobingelement, continuous movement, etc.). The visual indication output may beconfigured according to a user profile preference. In an example, audiospoken by the recipient in response to the original message may beoutput by the speech-controlled device 110 a as soon as the visualindication is no longer output. Optionally, in response to receiving thesignal, the speech-controlled device 110 a may output a tactile and/oran audible indication (illustrated as 828). The tactile indication mayinclude the speech-controlled device 110 a vibrating and/or a remotedevice in communication with the speech-controlled device 110 a (e.g., asmart watch) vibrating. The remote device and the speech-controlleddevice 110 a may be in communication by being located in a single tableof user devices associated with a user profile. The audible indicationmay include computer generated/TTS generated speech and/or usergenerated speech corresponding to, for example, “John Smith isresponding to your message” or “John Smith is speaking.” The audibleindication, like the tactile indication, may be output by thespeech-controlled device 110 a, a remote microphone array, and/or aremote device (e.g., a smart watch). The remote device, the microphonearray, and the speech-controlled device 110 a may be in communication bybeing located in a single table of user devices associated with a userprofile.

FIG. 9 illustrates the output of signaling via user interfaces ofspeech-controlled devices. A speech-controlled device 110 a capturesspoken audio including a wakeword portion and recipient information(illustrated as 902). The speech-controlled device 110 a converts thecaptures recipient information audio into audio data and sends the audiodata to the server 120 (illustrated as 904). Alternatively, thespeech-controlled device 110 a may send audio data corresponding to boththe wakeword portion and the recipient information to the server 120. Inthis example, the server 120 may isolate the recipient information audiodata from the wakeword portion audio data, and discard the wakewordportion audio data. The server 120 may perform speech processing (e.g.,ASR and NLU) on the recipient information audio data (illustrated as906). For example, the server 120 may perform ASR on the recipientinformation audio data to create recipient information text data, andmay perform NLU on the recipient information text data to identify therecipient name. If the speech-controlled device 110 a from which thereceived audio data originated is associated with multiple users, theserver 120 may perform various processes to determine which user spokethe wakeword portion and recipient information audio (illustrated as908).

Using the speech-processed recipient information audio data and knowingthe speaker of the recipient information audio, the server 120determines a device of the recipient, to send future data to, using auser profile associated with the speaker of the recipient informationaudio (illustrated as 910). If the recipient is associated with only onedevice in the user profile, that is the device to which data will besent. If the recipient is associated with multiple devices in the userprofile, various information may be used to determine which recipientdevice to send data to. For example, a physical location of therecipient may be determined, and data may be sent to the device mostproximate to the recipient. In another example, it may be determinedwhich device of the recipient is presently in use, and data may be sentto the device presently in use. In yet another example, it may bedetermined which device of the recipient is presently in use, and datamay be sent to a second device most proximate to the device presently inuse. In another example, the device determined by the server 120 (i.e.,the device to which future data will be sent) may be a distributordevice (e.g., a router), with the distributor device determining whichof multiple devices of the recipient to send data to.

The server 120 sends a signal indicated a message is forthcoming to thedetermined device of the recipient (illustrated as 912). The signal maybe sent to the recipient device while the server 120 sends messagecontent text data to a TTS component. For purposes of illustration, thedetermined device of the recipient may be the speech-controlled device110 b. The speech-controlled device 110 b then outputs an indicationrepresenting a message is forthcoming (illustrated as 914). Theindication output by the speech-controlled device may be a visualindication, audible indication, and/or tactile indication as describedherein.

The speech-controlled device 110 a of the message sender also capturesspoken audio including message content (illustrated as 916). Thespeech-controlled device 110 a converts the message content spoken audiointo audio data, and sends the message content audio data to the server120 (illustrated as 918). In an example, the speech-controlled device110 b outputs the indication while the speech-controlled device 110 acaptures the message content audio and while the server 120 receives themessage content audio from the speech-controlled device 110 a. Theserver 120 may send the message content audio data to the previouslydetermined recipient device (illustrated as 920), which outputs audioincluding the message content (illustrated as 922). Alternatively, theserver 120 may perform processes as described herein above with respectto step 910 to determine which recipient device to send the messagecontent audio data to. Thus, it should be appreciated that the recipientdevice that outputs the indication representing the message isforthcoming and the recipient device that outputs the message contentmay be the same device or may be different devices depending upon thesituation.

FIGS. 10A through 10C illustrate examples of a visual indicator asdiscussed herein. The visual indication may be output via a light ring1002 of the speech-controlled device 110. The light ring 1002 may belocated anywhere on the speech-controlled device 110 that enablesadequate viewing by a user of the speech-controlled device 110.Different colors may be output via the light ring 1002 depending uponthe message to be communicated. For example, the light ring 1002 mayemit a green light to indicate the message is or will be sent to arecipient device. In another example, the light ring 1002 may emit ablue light to indicate the recipient device is detecting or capturingspoken audio. It should also be appreciated that the light ring 1002 mayemit different shades of a single color to communicate differentmessages. For example, the light ring (illustrated as 1002 a in FIG.10A) may output a dark shade of a color to represent a first message,the light ring (illustrated as 1002 b in FIG. 10B) may output a mediumshade of a color to represent a second message, and the light ring(illustrated as 1002 c in FIG. 10C) may output a light shade of a colorto represent a third message. While three shades are illustrated, oneskilled in the art should appreciate that more or less than three shadesof a color may be implemented depending upon how many different messagesare to be communicated. Further, while the visual indicator examples ofFIGS. 10A through 10C may be static, they may also appear to move insome manner. For example, the visual indicators may blink, strobe, orcontinuously move around/along a surface of the device 110.

FIGS. 11A and 11B illustrate a motion indication as described herein. Asillustrated, the light ring 1002 may be configured to look as if aportion of the light ring 1002 is moving about the speech-controlleddevice 110. While not illustrated, it should also be appreciated thatthe light ring 1002 and/or the LED 1202/1204 may be configured to blink,strobe, etc.

FIG. 12 illustrates another visual indication as described herein.According to FIG. 11, the static visual indication may be output via anLED 1202/1204 or some other like light generating device. The LED1202/1204 may be located anywhere on the speech-controlled device 110that enables adequate viewing by a user of the speech-controlled device110. Different colors may be output via the LED 1202/1204 depending uponthe message to be communicated. For example, the LED 1202/1204 may emita green light to indicate the message is or will be sent to a recipientdevice. In another example, the LED 1202/1204 may emit a blue light toindicate the recipient device is detecting or capturing spoken audio. Itshould also be appreciated that the LED 1202/1204 may emit differentshades of a single color to communicate different messages. For example,the LED 1202/1204 may output a dark shade of a color to represent afirst message, a medium shade of a color to represent a second message,and a light shade of a color to represent a third message. While threeshades are described, one skilled in the art should appreciate that moreor less than three shades of a color may be implemented depending uponhow many different messages are to be communicated. It should beappreciated that both the light ring 1002 and the LED 1202/1204 may beimplemented on the same speech-controlled device 110, and that differentvariations of the indications described (and others) may be used.

While visual indicators are discussed above as examples of indicators,other indicators such as audio indicators, haptic indicators, etc., maybe used to indicate an incoming message, reply being spoken, etc.

FIG. 13 is a block diagram conceptually illustrating a user device 110(for example speech-controlled devices 110 a and 110 b as hereindescribed) that may be used with the described system. FIG. 14 is ablock diagram conceptually illustrating example components of a remotedevice, such as a remote server 120 that may assist with ASR processing,NLU processing, or command processing. Multiple such servers 120 may beincluded in the system, such as one server(s) 120 for performing ASR,one server(s) 120 for performing NLU, etc. In operation, each of thesedevices (or groups of devices) may include computer-readable andcomputer-executable instructions that reside on the respective device(110/120), as will be discussed further below.

Each of these devices (110/120) may include one or morecontrollers/processors (1304/1404), that may each include a centralprocessing unit (CPU) for processing data and computer-readableinstructions, and a memory (1306/1406) for storing data and instructionsof the respective device. The memories (1306/1406) may individuallyinclude volatile random access memory (RAM), non-volatile read onlymemory (ROM), non-volatile magnetoresistive (MRAM) and/or other types ofmemory. Each device may also include a data storage component(1308/1408), for storing data and controller/processor-executableinstructions. Each data storage component may individually include oneor more non-volatile storage types such as magnetic storage, opticalstorage, solid-state storage, etc. Each device may also be connected toremovable or external non-volatile memory and/or storage (such as aremovable memory card, memory key drive, networked storage, etc.)through respective input/output device interfaces (1302/1402).

Computer instructions for operating each device (110/120) and itsvarious components may be executed by the respective device'scontroller(s)/processor(s) (1304/1404), using the memory (1306/1406) astemporary “working” storage at runtime. A device's computer instructionsmay be stored in a non-transitory manner in non-volatile memory(1306/1406), storage (1308/1408), or an external device(s).Alternatively, some or all of the executable instructions may beembedded in hardware or firmware on the respective device in addition toor instead of software.

Each device (110/120) includes input/output device interfaces(1302/1402). A variety of components may be connected through theinput/output device interfaces (1302/1402), as will be discussed furtherbelow. Additionally, each device (110/120) may include an address/databus (1324/1424) for conveying data among components of the respectivedevice. Each component within a device (110/120) may also be directlyconnected to other components in addition to (or instead of) beingconnected to other components across the bus (1324/1424).

Referring to the device 110 of FIG. 13, the device 110 may include adisplay 1318, which may comprise a touch interface 1019 configured toreceive limited touch inputs. Or the device 110 may be “headless” andmay primarily rely on spoken commands for input. As a way of indicatingto a user that a connection between another device has been opened, thedevice 110 may be configured with a visual indicator, such as an LED orsimilar component (not illustrated), that may change color, flash, orotherwise provide visual indications by the device 110. The device 110may also include input/output device interfaces 1302 that connect to avariety of components such as an audio output component such as aspeaker 101, a wired headset or a wireless headset (not illustrated) orother component capable of outputting audio. The device 110 may alsoinclude an audio capture component. The audio capture component may be,for example, a microphone 103 or array of microphones, a wired headsetor a wireless headset (not illustrated), etc. The microphone 103 may beconfigured to capture audio. If an array of microphones is included,approximate distance to a sound's point of origin may be determined byacoustic localization based on time and amplitude differences betweensounds captured by different microphones of the array. The device 110(using microphone 103, wakeword detection module 220, ASR module 250,etc.) may be configured to determine audio data corresponding todetected audio data. The device 110 (using input/output deviceinterfaces 1002, antenna 1014, etc.) may also be configured to transmitthe audio data to server 120 for further processing or to process thedata using internal components such as a wakeword detection module 220.

For example, via the antenna(s) 1314, the input/output device interfaces1302 may connect to one or more networks 199 via a wireless local areanetwork (WLAN) (such as WiFi) radio, Bluetooth, and/or wireless networkradio, such as a radio capable of communication with a wirelesscommunication network such as a Long Term Evolution (LTE) network, WiMAXnetwork, 3G network, etc. A wired connection such as Ethernet may alsobe supported. Through the network(s) 199, the speech processing systemmay be distributed across a networked environment.

The device 110 and/or server 120 may include an ASR module 250. The ASRmodule in device 110 may be of limited or extended capabilities. The ASRmodule 250 may include the language models 254 stored in ASR modelstorage component 252, and an ASR module 250 that performs the automaticspeech recognition process. If limited speech recognition is included,the ASR module 250 may be configured to identify a limited number ofwords, such as keywords detected by the device, whereas extended speechrecognition may be configured to recognize a much larger range of words.

The device 110 and/or server 120 may include a limited or extended NLUmodule 260. The NLU module in device 110 may be of limited or extendedcapabilities. The NLU module 260 may comprising the name entityrecognition module 262, the intent classification module 264 and/orother components. The NLU module 260 may also include a stored knowledgebase and/or entity library, or those storages may be separately located.

The device 110 and/or server 120 may also include a command processor290 that is configured to execute commands/functions associated with aspoken command as described above.

The device 110 may include a wakeword detection module 220, which may bea separate component or may be included in an ASR module 250. Thewakeword detection module 220 receives audio signals and detectsoccurrences of a particular expression (such as a configured keyword) inthe audio. This may include detecting a change in frequencies over aspecific period of time where the change in frequencies results in aspecific audio signature that the system recognizes as corresponding tothe keyword. Keyword detection may include analyzing individualdirectional audio signals, such as those processed post-beamforming ifapplicable. Other techniques known in the art of keyword detection (alsoknown as keyword spotting) may also be used. In some embodiments, thedevice 110 may be configured collectively to identify a set of thedirectional audio signals in which the wake expression is detected or inwhich the wake expression is likely to have occurred.

The wakeword detection module 220 receives captured audio and processesthe audio (for example, using model(s) 232) to determine whether theaudio corresponds to particular keywords recognizable by the device 110and/or system 100. The storage 1308 may store data relating to keywordsand functions to enable the wakeword detection module 220 to perform thealgorithms and methods described above. The locally stored speech modelsmay be preconfigured based on known information, prior to the device 110being configured to access the network by the user. For example, themodels may be language and/or accent specific to a region where the userdevice is shipped or predicted to be located, or to the userhimself/herself, based on a user profile, etc. In an aspect, the modelsmay be pre-trained using speech or audio data of the user from anotherdevice. For example, the user may own another user device that the useroperates via spoken commands, and this speech data may be associatedwith a user profile. The speech data from the other user device may thenbe leveraged and used to train the locally stored speech models of thedevice 110 prior to the user device 110 being delivered to the user orconfigured to access the network by the user. The wakeword detectionmodule 220 may access the storage 1308 and compare the captured audio tothe stored models and audio sequences using audio comparison, patternrecognition, keyword spotting, audio signature, and/or other audioprocessing techniques.

As noted above, multiple devices may be employed in a single speechprocessing system. In such a multi-device system, each of the devicesmay include different components for performing different aspects of thespeech processing. The multiple devices may include overlappingcomponents. The components of the devices 110 and server 120, asillustrated in FIGS. 13 and 14, are exemplary, and may be located as astand-alone device or may be included, in whole or in part, as acomponent of a larger device or system.

To create output speech, the server 120 may be configured with atext-to-speech (“TTS”) module 1410 that transforms text data into audiodata representing speech. The audio data may then be sent to the device110 for playback to the user, thus creating the output speech. The TTSmodule 1410 may include a TTS storage for converting the input text intospeech. The TTS module 1410 may include its owncontroller(s)/processor(s) and memory or may use thecontroller/processor and memory of the server(s) 120 or other device,for example. Similarly, the instructions for operating the TTS module1410 may be located within the TTS module 1410, within the memory and/orstorage of the server(s) 120, or within an external device.

Text input into a TTS module 1410 may be processed to perform textnormalization, linguistic analysis, and linguistic prosody generation.During text normalization, the TTS module 1410 processes the text inputand generates standard text, converting such things as numbers,abbreviations (such as Apt., St., etc.), and symbols ($, %, etc.) intothe equivalent of written out words.

During linguistic analysis the TTS module 1410 analyzes the language inthe normalized text to generate a sequence of phonetic unitscorresponding to the input text. This process may be referred to asphonetic transcription. Phonetic units include symbolic representationsof sound units to be eventually combined and output by the system 100 asspeech. Various sound units may be used for dividing text for purposesof speech synthesis. The TTS module 1410 may process speech based onphonemes (individual sounds), half-phonemes, di-phones (the last half ofone phoneme coupled with the first half of the adjacent phoneme),bi-phones (two consecutive phonemes), syllables, words, phrases,sentences, or other units. Each word may be mapped to one or morephonetic units. Such mapping may be performed using a languagedictionary stored by the system 100, for example in the TTS storage. Thelinguistic analysis performed by the TTS module 1410 may also identifydifferent grammatical components such as prefixes, suffixes, phrases,punctuation, syntactic boundaries, or the like. Such grammaticalcomponents may be used by the TTS module 1410 to craft a naturalsounding audio waveform output. The language dictionary may also includeletter-to-sound rules and other tools that may be used to pronouncepreviously unidentified words or letter combinations that may beencountered by the TTS module 1410. Generally, the more informationincluded in the language dictionary, the higher quality the speechoutput.

Based on the linguistic analysis, the TTS module 1410 may then performlinguistic prosody generation where the phonetic units are annotatedwith desired prosodic characteristics, also called acoustic features,which indicate how the desired phonetic units are to be pronounced inthe eventual output speech. During this stage the TTS module 1410 mayconsider and incorporate any prosodic annotations that accompanied thetext input. Such acoustic features may include pitch, energy, duration,and the like. Application of acoustic features may be based on prosodicmodels available to the TTS module 1410. Such prosodic models indicatehow specific phonetic units are to be pronounced in certaincircumstances. A prosodic model may consider, for example, a phoneme'sposition in a syllable, a syllable's position in a word, a word'sposition in a sentence, phrase, or paragraph, neighboring phoneticunits, etc. As with the language dictionary, prosodic models with moreinformation may result in higher quality speech output than prosodicmodels with less information. As can be appreciated, when a largerportion of a textual work is made available to the TTS module 1410, theTTS module 1410 may assign more robust and complex prosodiccharacteristics that vary across the portion, thus making the portionsound more human, resulting in higher quality audio output.

The TTS module 1410 may generate a symbolic linguistic representation,which may include a sequence of phonetic units annotated with prosodiccharacteristics. This symbolic linguistic representation may then beconverted into an audio waveform of speech for output to an audio outputdevice (such as a microphone) and eventually to a user. The TTS module1410 may be configured to convert the input text into high-qualitynatural-sounding speech in an efficient manner. Such high-quality speechmay be configured to sound as much like a human speaker as possible, ormay be configured to be understandable to a listener without attempts tomimic a specific human voice.

The TTS module 1410 may perform speech synthesis using one or moredifferent methods. In one method of synthesis called unit selection,described further below, the TTS module 1410 matches the symboliclinguistic representation against a database of recorded speech, such asa database of a voice corpus. The TTS module 1410 matches the symboliclinguistic representation against spoken audio units in the database.Matching units are selected and concatenated together to form a speechoutput. Each unit includes an audio waveform corresponding with aphonetic unit, such as a short .wav file of the specific sound, alongwith a description of the various acoustic features associated with the.wav file (such as its pitch, energy, etc.), as well as otherinformation, such as where the phonetic unit appears in a word,sentence, or phrase, the neighboring phonetic units, etc. Using all theinformation in the unit database, the TTS module 1410 may match units(for example in a unit database) to the input text to create a naturalsounding waveform. The unit database may include multiple examples ofphonetic units to provide the system 100 with many different options forconcatenating units into speech. One benefit of unit selection is that,depending on the size of the database, a natural sounding speech outputmay be generated. As described above, the larger the unit database ofthe voice corpus, the more likely the system will be able to constructnatural sounding speech.

In another method of synthesis, called parametric synthesis, parameterssuch as frequency, volume, and noise are varied by the TTS module 1410to create an artificial speech waveform output. Parametric synthesis mayuse an acoustic model and various statistical techniques to match asymbolic linguistic representation with desired output speechparameters. Parametric synthesis may include the ability to be accurateat high processing speeds, as well as the ability to process speechwithout large databases associated with unit selection, but alsotypically produces an output speech quality that may not match that ofunit selection. Unit selection and parametric techniques may beperformed individually or combined together and/or combined with othersynthesis techniques to produce speech audio output.

Parametric speech synthesis may be performed as follows. The TTS module1410 may include an acoustic model, or other models, which may convert asymbolic linguistic representation into a synthetic acoustic waveform ofthe text input based on audio signal manipulation. The acoustic modelincludes rules that may be used to assign specific audio waveformparameters to input phonetic units and/or prosodic annotations. Therules may be used to calculate a score representing a likelihood that aparticular audio output parameter(s) (such as frequency, volume, etc.)corresponds to the portion of the input symbolic linguisticrepresentation.

As illustrated in FIG. 15 multiple devices (120, 110, 110 c-110 f) maycontain components of the system 100 and the devices may be connectedover a network 199. Network 199 may include a local or private networkor may include a wide network such as the Internet. Devices may beconnected to the network 199 through either wired or wirelessconnections. For example, a speech controlled device 110, a tabletcomputer 110 e, a smart phone 110 c, a smart watch 110 d, and/or avehicle 110 f may be connected to the network 199 through a wirelessservice provider, over a WiFi or cellular network connection or thelike. Other devices are included as network-connected support devices,such as a server 120, application developer devices, or others. Thesupport devices may connect to the network 199 through a wiredconnection or wireless connection. Networked devices 110 may captureaudio using one-or-more built-in or connected microphones 103 or audiocapture devices, with processing performed by ASR, NLU, or othercomponents of the same device or another device connected via network199, such as an ASR 250, NLU 260, etc. of one or more servers 120.

The concepts disclosed herein may be applied within a number ofdifferent devices and computer systems, including, for example,general-purpose computing systems, speech processing systems, anddistributed computing environments.

The above aspects of the present disclosure are meant to beillustrative. They were chosen to explain the principles and applicationof the disclosure and are not intended to be exhaustive or to limit thedisclosure. Many modifications and variations of the disclosed aspectsmay be apparent to those of skill in the art. Persons having ordinaryskill in the field of computers and speech processing should recognizethat components and process steps described herein may beinterchangeable with other components or steps, or combinations ofcomponents or steps, and still achieve the benefits and advantages ofthe present disclosure. Moreover, it should be apparent to one skilledin the art, that the disclosure may be practiced without some or all ofthe specific details and steps disclosed herein.

Aspects of the disclosed system may be implemented as a computer methodor as an article of manufacture such as a memory device ornon-transitory computer readable storage medium. The computer readablestorage medium may be readable by a computer and may compriseinstructions for causing a computer or other device to perform processesdescribed in the present disclosure. The computer readable storage mediamay be implemented by a volatile computer memory, non-volatile computermemory, hard drive, solid-state memory, flash drive, removable diskand/or other media. In addition, components of one or more of themodules and engines may be implemented as in firmware or hardware, suchas the acoustic front end 256, which comprise among other things, analogand/or digital filters (e.g., filters configured as firmware to adigital signal processor (DSP)).

As used in this disclosure, the term “a” or “one” may include one ormore items unless specifically stated otherwise. Further, the phrase“based on” is intended to mean “based at least in part on” unlessspecifically stated otherwise.

What is claimed is:
 1. A system comprising: at least one processor; andmemory including instructions operable to be executed by the at leastone processor to configure the system to: receive, from a first device,input audio data including message content; send, to a second deviceoperating in a first mode and in a synchronous communication with thefirst device, output audio data corresponding to the message content;receive, from the second device, an indication that the second devicehas detected a reply to the output audio data and is operating in asecond mode corresponding to audio detection; generate, in response tothe indication, a notification that the second device is operating inthe second mode; cause the first device to output the notification; andcause the first device to enter into a third mode of operation to outputthe reply.
 2. The system of claim 1, wherein the memory furthercomprises instructions that, when executed by the at least oneprocessor, further configure the system to: identify the second deviceusing a user profile associated with the first device.
 3. The system ofclaim 2, wherein the memory further comprises instructions that, whenexecuted by the at least one processor, further configure the system to:determine a setting in the user profile indicates the synchronouscommunication is permitted with the second device.
 4. The system ofclaim 1, wherein the memory further comprises instructions that, whenexecuted by the at least one processor, further configure the system to:identify the second device based at least in part on processing of theinput audio data.
 5. The system of claim 1, wherein the memory furthercomprises instructions that, when executed by the at least oneprocessor, wherein the notification includes a visual indicatorrepresenting that the second device is operating in the second mode. 6.The system of claim 1, wherein the memory further comprises instructionsthat, when executed by the at least one processor, further configure thesystem to: determine, by the first device, a wake command.
 7. The systemof claim 1, wherein the memory further comprises instructions that, whenexecuted by the at least one processor, further configure the system to,prior to receipt of the indication: cause the second device to outputaudio corresponding to the output audio data.
 8. The system of claim 1,wherein the synchronous communication corresponds to a Voice overInternet Protocol (VoIP) communication.
 9. The system of claim 1,wherein the memory further comprises instructions that, when executed bythe at least one processor, further configure the system to: process theinput audio data to determine the message content; and configure theoutput audio data to include the message content.
 10. Acomputer-implemented method comprising: receiving, from a first device,input audio data including message content; sending, to a second deviceoperating in a first mode and in a synchronous communication with thefirst device, output audio data corresponding to the message content;receiving, from the second device, an indication that the second devicehas detected a reply to the output audio data and is operating in asecond mode corresponding to audio detection; generating, in response tothe indication, a notification that the second device is operating inthe second mode; causing the first device to output the notification;and causing the first device to enter into a third mode of operation tooutput the reply.
 11. The computer-implemented method of claim 10,further comprising: identifying the second device using a user profileassociated with the first device.
 12. The computer-implemented method ofclaim 11, further comprising: determining a setting in the user profileindicates the synchronous communication is permitted with the seconddevice.
 13. The computer-implemented method of claim 10, furthercomprising: identifying the second device based at least in part onprocessing of the input audio data.
 14. The computer-implemented methodof claim 10, wherein the notification includes a visual indicatorrepresenting that the second device is operating in the second mode. 15.The computer-implemented method of claim 10, further comprising:determining, by the first device, a wake command.
 16. Thecomputer-implemented method of claim 10, further comprising, prior toreceipt of the indication: causing the second device to output audiocorresponding to the output audio data.
 17. The computer-implementedmethod of claim 10, wherein the synchronous communication corresponds toa Voice over Internet Protocol (VoIP) communication.
 18. Thecomputer-implemented method of claim 10, further comprising: processingthe input audio data to determine the message content; and configuringthe output audio data to include the message content.